Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datadog scaler multi query #1

Closed
wants to merge 9 commits into from
Closed

Datadog scaler multi query #1

wants to merge 9 commits into from

Conversation

dgibbard-cisco
Copy link
Owner

@dgibbard-cisco dgibbard-cisco commented Jul 26, 2022 •
edited
Loading

Add support for Datadog Metric API responses which container more than one Series (ie. when submitting multiple queries, comma-seperated), and allow for aggregation of results.
This allows for scaling on multiple datadog queries to be combined into a single API query, reducing Rate Limiting errors.

Checklist

  • Commits are signed with Developer Certificate of Origin (DCO - learn more)
  • Tests have been added
  • A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
  • A PR is opened to update the documentation on (repo) (if applicable)
  • Changelog has been updated and is aligned with our changelog requirements

Fixes #

Relates to #

dgibbard-cisco and others added 9 commits July 26, 2022 12:17
@dgibbard-cisco
Datadog Scaler: Add support for multi-query results
fc18173 
Signed-off-by: Darren Gibbard <dgibbard@cisco.com>
@dgibbard-cisco
Datadog Scaler: Add support for multi-query results
c3d9347 
Signed-off-by: Darren Gibbard <dgibbard@cisco.com>
@dgibbard-cisco
Removed unneeded changes
62137a0 
Signed-off-by: Darren Gibbard <dgibbard@cisco.com>
@dgibbard-cisco
Add queryAggregator meta + utils + tests
cc08baf 
Signed-off-by: Darren Gibbard <dgibbard@cisco.com>
@dgibbard-cisco
Move operation methods to scaler code for simplicity
a121b30
@dgibbard-cisco
debug
37c4ded 
Signed-off-by: Darren Gibbard <dgibbard@cisco.com>
@dgibbard-cisco
Working, but not working
f08cc53
@dgibbard-cisco
Merge branch 'main' into datadog-scaler-multi-query
74f918d 
Signed-off-by: dgibbard-cisco <57677847+dgibbard-cisco@users.noreply.github.com>
@dgibbard-cisco
Update CHANGELOG.md
29d1c51 
Signed-off-by: dgibbard-cisco <57677847+dgibbard-cisco@users.noreply.github.com>
@dgibbard-cisco
Copy link
Owner Author

dgibbard-cisco commented Aug 10, 2022 •
edited
Loading

Have been testing this locally, added debug all over the place to track the values and it works; ie. the values are retrieved from Datadog, and aggregated as desired... but the HPA reports:

unable to get external metric dlp-scanner/s0-datadog-per_second(sum-http-requests/\u0026LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: myapp,},MatchExpressions:[]LabelSelectorRequirement{},}:
unable to fetch metrics from external metrics API: no matching metrics found

the ScaledObject status is:

Status:
    Conditions:
        Message:  ScaledObject is defined correctly and is ready for scaling
        Reason:   ScaledObjectReady
        Status:   True
        Type:     Ready
        Message:  Scaling is performed because triggers are active
        Reason:   ScalerActive
        Status:   True
        Type:     Active
        Message:  No fallbacks are active on this scaled object
        Reason:   NoFallbackFound
        Status:   False
        Type:     Fallback
    External Metric Names:
        s0-datadog-per_second(sum-http-requests
    Health:
        s0-datadog-per_second(sum-http-requests:
            Number Of Failures:  35
            Status:              Failing

Events only shows:

Normal   ScaledObjectReady  5m35s (x4 over 8m52s)  keda-operator  ScaledObject is ready for scaling

No errors...

@dgibbard-cisco
Copy link
Owner Author

dgibbard-cisco commented Aug 10, 2022 •
edited
Loading

Ah found it:

keda_metrics_adapter/datadog_scaler "msg"="error getting metrics from Datadog" "error"="query returned more than 1 series; modify the query to return only 1 series"

Probably helps when i'm using the correct image for testing the keda metrics server too 🤦

@dgibbard-cisco dgibbard-cisco deleted the datadog-scaler-multi-query branch August 12, 2022 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@dgibbard-cisco